Maintaining Service Performance During a Cloud Upgrade

ABSTRACT

Systems, methods, and computer storage media for upgrading a domain in a distributed computing environment are provided. Upgrading of the domain includes preparing for the upgrade, upgrading, and finalizing the upgrade. The preparation of the domain includes ensuring predefined quantities of role instances are available in domains other than the upgrade domain. The preparation also includes ensuring that a predefined number of extent replicas are available in domains other than the upgrade domain. The preparation may also include checkpointing partitions within the upgrade domain to facilitate faster loading once transferred to a domain other than the upgrade domain. The finalization may include allowing nodes within the upgrade domain to resume functionality that was suspended during the upgrade.

BACKGROUND

Typically, when a service in a distributed computing environment isupgraded, nodes operating the service are taken offline to facilitatethe upgrade process. When a node is taken offline, the data and servicesthat are associated with the node may be unavailable to clients of thedistributed computing environment. During the upgrade of a node, thedistributed computing environment traditionally compensates for theoffline node as if a failure of the node occurred.

SUMMARY

Embodiments of the present invention relate to systems, methods andcomputer storage media for upgrading a domain in a distributed computingenvironment. Upgrading of the domain may include preparing for theupgrade, upgrading, and finalizing the upgrade. The preparation of thedomain includes ensuring predefined quantities of role instances areavailable in domains other than the upgrade domain. The preparation alsoincludes ensuring that a predefined number of extent replicas areavailable in domains other than the upgrade domain. The preparation mayalso include checkpointing partitions within the upgrade domain tofacilitate faster loading of the partition once transferred to a domainother than the upgrade domain. The finalization may include allowingnodes within the upgrade domain to resume functionality suspended duringthe upgrade.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Illustrative embodiments of the present invention are described indetail below with reference to the attached drawing figures, which areincorporated by reference herein and wherein:

FIG. 1 depicts an exemplary computing device suitable for implementingembodiments of the present invention;

FIG. 2 depicts an exemplary distributed computing environment in whichembodiments of the present invention may be employed;

FIG. 3 depicts an exemplary upgrade domain in accordance withembodiments of the present invention;

FIG. 4 depicts a manager in accordance with an embodiment of the presentinvention;

FIG. 5 depicts a high-level method for upgrading a domain in accordancewith embodiments of the present invention;

FIG. 6 depicts a diagram illustrating a sequence for upgrading ofdomains in different groups that are not geo-related in accordance withembodiments of the present invention;

FIG. 7 depicts a method for upgrading a service in a distributedcomputing environment in accordance with embodiments of the presentinvention;

FIG. 8 depicts a method for upgrading a service in a distributedcomputing environment in accordance with embodiments of the presentinvention; and

FIG. 9 depicts a method for upgrading a service in a distributedcomputing environment in accordance with embodiments of the presentinvention.

DETAILED DESCRIPTION

The subject matter of embodiments of the present invention is describedwith specificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies.

Embodiments of the present invention relate to methods and computerstorage media for upgrading a domain in a distributed computingenvironment. A domain is a set of nodes from a cluster of nodes. Thecluster of nodes may be divided up into “N” different domain. Upgradingof the domain includes preparing for the upgrade, upgrading, andfinalizing the upgrade. The preparation of the domain includes ensuringpredefined quantities of role instances are available in domains otherthan the upgrade domain. The preparation also includes ensuring that apredefined number of extent replicas are available in domains other thanthe upgrade domain. The preparation may also include checkpointingpartitions within the upgrade domain to facilitate faster loading oncetransferred to a domain other than the upgrade domain. The finalizationmay include allowing nodes within the upgrade domain to resumefunctionality that may have been suspended during the upgrade.

Accordingly, in one aspect, the present invention provides computerstorage media having computer-executable instructions embodied thereon,that when executed by a computing system having a processor and memory,cause the computing system to perform a method for upgrading a servicein a distributed computing environment. The method includes selecting anupgrade domain within the distributed computing environment in which toupgrade the service. Further, the method includes offloading data, inanticipation of the upgrade, from a node within the upgrade domain toone or more nodes in a domain other than the upgrade domain. The methodalso includes notifying the distributed computing environment that theupgrade domain is unavailable as a result of being upgraded. The methodalso includes upgrading the service in the upgrade domain. Upgrading mayinclude updating BIOS settings, an operating system, a service software,service software settings, security information for an operating system,and security information for a service software. The method additionallyincludes loading the data to the node. The method further includesnotifying the distributed computing environment that the upgrade domainis available.

In another aspect, the present invention provides a computer-implementedmethod for upgrading a service in a distributed computing environment.The method includes preparing an upgrade domain of the distributedcomputing environment for an upgrade. The preparing includes identifyinga role operating in the upgrade domain. Further the preparing includesdetermining an available quantity of instances of the role in domainsother than the upgrade domain is above a predefined threshold.Additionally, the preparing includes checkpointing a partition in theupgrade domain. Further, the preparing includes transferring anassignment for the partition to a domain other than the upgrade domain.The preparing also includes identifying an extent in the upgrade domain.The preparing also includes determining an available quantity ofreplicas of the extent in domains other than the upgrade domain is abovea predefined threshold. The method includes upgrading the upgrade domainwith the upgrade.

A third aspect of the present invention provides a method for upgradinga service in a distributed computing environment. The method includespreparing an upgrade domain that is comprised of a plurality of serversof the distributed computing environment for an upgrade. The preparingincludes determining a number of instances of a role in one or moredomains other than the upgrade domain are above a predefined threshold.The role is responsible for at least some functionality associated withthe service. The preparing also includes checkpointing, to facilitatefaster loading upon transfer, a partition served from a partition serverin the upgrade domain. The preparing additionally includes transferringan assignment to the partition to a domain other than the upgradedomain. The preparing also includes determining a number of replicas ofan extent in one or more domains other than the upgrade domain are abovea predefined threshold. The preparing additionally includes preventing anew extent instance from being created in the upgrade domain. Further,preparing includes notifying the distributed computing environment thatextents are unavailable from the upgrade domain. The method includesupgrading the upgrade domain with the upgrade. The method also includesfinalizing the upgrade in the upgrade domain. Finalizing includesauthorizing the partition server to serve one or more partitions.Finalizing also includes allowing a new extent instance to be created inthe upgrade domain. Finalizing additionally includes notifying thedistributed computing environment that one or more extent instances areavailable from the upgrade domain.

Having briefly described an overview of embodiments of the presentinvention, an exemplary operating environment suitable for implementingembodiments hereof is described below.

Referring to the drawings in general, and initially to FIG. 1 inparticular, an exemplary operating system suitable for implementingembodiments of the present invention is shown and designated generallyas computing device 100. Computing device 100 is but one example of asuitable computing environment and is not intended to suggest anylimitation as to the scope of use or functionality of the invention.Neither should the computing device 100 be interpreted as having anydependency or requirement relating to any one or combination ofmodules/components illustrated.

Embodiments may be described in the general context of computer code ormachine-useable instructions, including computer-executable instructionssuch as program modules, being executed by a computer or other machine,such as a personal data assistant or other handheld device. Generally,program modules including routines, programs, objects, modules, datastructures, and the like, refer to code that performs particular tasksor implements particular abstract data types. Embodiments may bepracticed in a variety of system configurations, including hand-helddevices, consumer electronics, general-purpose computers, specialtycomputing devices, etc. Embodiments may also be practiced in distributedcomputing environments where tasks are performed by remote-processingdevices that are linked through a communications network.

With continued reference to FIG. 1, computing device 100 includes a bus110 that directly or indirectly couples the following devices: memory112, one or more processors 114, one or more presentation component 116,input/output (I/O) ports 118, I/O components 120, and an illustrativepower supply 122. Bus 110 represents what may be one or more busses(such as an address bus, data bus, or combination thereof). Although thevarious blocks of FIG. 1 are shown with lines for the sake of clarity,in reality, delineating various modules is not so clear, andmetaphorically, the lines would more accurately be grey and fuzzy. Forexample, one may consider a presentation module such as a display deviceto be an I/O module. Also, processors have memory. The inventors hereofrecognize that such is the nature of the art, and reiterate that thediagram of FIG. 1 is merely illustrative of an exemplary computingdevice that can be used in connection with one or more embodiments.Distinction is not made between such categories as “workstation,”“server,” “laptop,” “hand-held device,” etc., as all are contemplatedwithin the scope of FIG. 1 and reference to “computer” or “computingdevice.”

Computing device 100 typically includes a variety of computer-readablemedia. By way of example, and not limitation, computer-readable mediamay comprise Random Access Memory (RAM); Read Only Memory (ROM);Electronically Erasable Programmable Read Only Memory (EEPROM); flashmemory or other memory technologies; CDROM, digital versatile disks(DVD) or other optical or holographic media; magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to encode desired information andbe accessed by computing device 100.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 100includes one or more processors that read data from various entitiessuch as memory 112 or I/O components 120. Presentation module(s) 116present data indications to a user or other device. Exemplarypresentation modules include a display device, speaker, printing module,vibrating module, and the like. I/O ports 118 allow computing device 100to be logically coupled to other devices including I/O components 120,some of which may be built in. Illustrative modules include amicrophone, joystick, game pad, satellite dish, scanner, printer,wireless device, and the like.

With reference to FIG. 2, a block diagram is provided illustrating anexemplary distributed computing system 200 in which embodiments of thepresent invention may be employed. It should be understood that this andother arrangements described herein are set forth only as examples.Other arrangements and elements (e.g., machines, components, computer,networks, interfaces, functions, orders, and grouping of functions,etc.) can be used in addition to or instead of those shown, and someelements may be omitted altogether. Further many of the elementsdescribed herein are functional entities that may be implemented asdiscrete or distributed components or in conjunction with othercomponents, and in any suitable combination and location. Variousfunctions described herein as being performed by one or more entitiesmay be carried out by hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory.

Among other components not shown, the system 200 may include a network202, a manager 204, an upgrade domain 1 206, and an upgrade domain 2214. The upgrade domain 1 206 may include a node A 208, a node B 210,and a node N 212. The upgrade domain 2, 214 may include a node X 216, anode Y 218, and a node M 220. Each of the components shown in FIG. 2 maybe any type of computing device, such as computing device 100 describedwith reference to FIG. 1, for example.

The components/nodes may communicate with each other via the network202, which may include, without limitation, one or more local areanetworks (LANs) and/or wide area networks (WANs). Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets, distributed computing networks, and the Internet.It should be understood that any number of nodes, fabric managers,upgrade domains, and networks may be employed within the system 200while staying within the scope of the present invention. Additionallyother components not shown may also be included within the system 200.

A distributed computing environment/system is a computing system that isdistributed, scalable, and accessible. Such a system may be referred toas a cloud computing system. Such a system is known to one of ordinaryskill in the art and will not be discussed in greater detail herein.

The manager 204 is functional to control one or more aspects associatedwith upgrading, accessing, or utilizing nodes of the distributedcomputing environment. For example, the manager 204, in an exemplaryembodiment, is responsible for facilitating the upgrading of a domain,such as the upgrade domain 1 206. The manager 204, in an exemplaryembodiment, is an operating system for one or more aspects of adistributed computing environment. For example, the manager 204 mayserve as a storage operating system that, like other operating systems,is responsible for controlling many core functionalities of a storagesystem. In this example, a storage system includes a number of nodesthat are used to store data in a distributed computing environment. Anexemplary manager is discussed hereinafter at FIG. 4.

In an embodiment, the manager 204 allows a service to be specified as aset of different roles, which are software components of the service. Aservice is able to specify what various roles are used, how manyinstances (e.g., copies) the service needs to have operating, and whatroles are allowed to communicate with one another. A service may alsospecify a number of domains to deploy the various role instances (e.g.,copies of the various software components). The fabric manager, in anexemplary embodiment, is responsible for deploying and maintaining thevarious role instances across the domains in compliance with the servicespecification.

The manager 204, among other responsibilities, may be responsible togroup one or more nodes into domains. For example, the manager 204 isresponsible for grouping the node A 208, the node B 210, and the node N212 into a common domain, the upgrade domain 1 206. Nodes of a storagesystem may be grouped into “#” (e.g., 1, 5, 10, 20, and 50) number ofdomains. The value of “#” affects various metrics associated withupgrading a service in a storage system. For example, the fewer numberof domains (i.e., greater number of nodes per domain) in which nodes areplaced, the less time that may be required to complete an update of theservice across all node (e.g., fewer total upgrade iterations may berequired). Conversely, the greater the number of domains (i.e., fewernumber of the nodes per domain) the more time may be required tocomplete an upgrade of a service, but the total number of nodes that areunavailable during the upgrade of any one domain is less. Therefore, inan exemplary embodiment, when only one domain is updated at a particulartime (i.e., the upgrade is done in series as opposed to in parallel),the fewer the number “#” domains, the faster the upgrade process acrossall nodes. But, with the fewer number “#” domains, a greater number ofnodes may be unavailable at a given time.

For example, the upgrade domain 1 206 and the upgrade domain 2 214, inan exemplary embodiment, are not upgraded in parallel (e.g.,simultaneously). Upgrading of domains in series facilitates maintainingdurability and reliability within a distributed computing environment.For example, if an upgrade to a service includes an error that causesfailure of one or more functionalities (e.g., roles) of the service, theservice may also fail. However, if the domains are upgraded in series,the non-updated domains may continue to function after an upgradeddomain fails. As will be discussed in more detail hereinafter at FIG. 6,domains of a distributed computing environment may be classified intodifferent geo-related groups. In this example, domains that are indifferent clusters that are in different geo-related groups may beupgraded in parallel to one another for reasons to be discussedhereinafter. A geo-related group may be comprised of a number of storageclusters. A storage cluster may be comprised of a number of domains. Adomain may be comprised of a number of nodes. Clusters within a commongeo-related group have a geo-replication dependency to one another inthe geo-related group.

A node is a computing device that provides functionality, such as aserver. In an exemplary embodiment, a node is a server having aprocessor and memory to facilitate the access and storage of data thatis used by one or more clients of a distributed computing environment.However, it is contemplated that each node is not a discrete device orphysical entity, but instead a common device or physical entity that mayinclude a number of different roles that provide various functionalityat multiple layers. Additional examples of nodes will be discussed ingreater detail at FIG. 3 hereinafter.

Accordingly, any number of components may be employed to achieve thedesired functionality within the scope of embodiments of the presentinvention. Although the various components of FIG. 2 are shown withlines for the sake of clarity, in reality, delineating variouscomponents is not so clear. Further, although some components of FIG. 2are depicted as single blocks, the depictions are exemplary in natureand in number and are not to be construed as limiting.

Turning to FIG. 3, FIG. 3 depicts an exemplary upgrade domain 300 inaccordance with embodiments of the present invention. The upgrade domain300 includes a front-end node 302, a partition node 306, and an extentnode 310. An exemplary service is a storage service. Such a service mayinclude three layers for various roles to operate within. For example,FIG. 3 illustrates a front-end layer 304, a partition layer 308, and astream layer 312.

The front-end layer 304 receives incoming requests for the storageservice. Additionally, the front-end layer 304 determines what partitionthe request references. Data stored in the storage system may be dividedinto partitions, which are slices of the data as a whole. The front-endlayer 304 may then forward the request to an appropriate partitionserver in the partition layer 308. In an exemplary embodiment, thefront-end layer 304 is comprised of front-end nodes, such as thefront-end node 302. For example, the front-end node 302 may be astateless node that serves as a role for the front-end layer 304. Afabric manager may evenly distribute front-end nodes across differentdomains of a distributed computing environment. In an exemplaryembodiment, a distributed computing environment may consist of one toone hundred front-end nodes that are dispersed evenly or as requiredacross all domains by a fabric manager.

The second layer, the partition layer 308, processes requests that havebeen forwarded by the front-end layer 304. For example, a receivedrequest may be processed at the partition layer 308 to a correspondingtable, blob, or queue-type partition. In this example, when the requestis a write request or a delete request, persistent data is stored in thestream layer 312. However, in this example, when the request is a readrequest, the request is served from either the memory at the partitionlayer 308 or read from the stream layer 312.

In an exemplary embodiment, the partition layer 308 is comprised ofhundreds of partition nodes, such as the partition node 306. In thisexample, each partition node serves a set of partitions. Further, in anexemplary embodiment, a fabric manager spreads the partition nodesevenly across the domains. However, it is contemplated that thepartition nodes may be dispersed by other methods across the variousdomains.

Additionally, not shown at FIG. 3, but nonetheless contemplated, thepartition layer 308 may also include a partition master role thatmanages partition servers and partitions of the partition layer 308. Inan exemplary embodiment, five to ten partition master roles are providedin a storage system. However, it is contemplated that fewer oradditional partition master roles may be implemented to achieveembodiment of the present invention. In an exemplary embodiment eachpartition manager role is located in a different domain by a fabricmanager.

The third layer, the stream layer 312 is a layer that providesreplication and persistent storage for data of a storage system. In anexemplary embodiment, data stored in a storage system is stored asstreams. Streams, in this example, consist of an ordered list ofextents. An extent is a block of data. Therefore, in this example, anordered list of extents defines a stream. Further, in an exemplaryembodiment, each unique extent is replicated at least three times in thedistributed computing environment. However, it is contemplated thatadditional or fewer extent replications are maintained to achieveembodiments of the present invention.

In an exemplary embodiment, the stream layer 312 is comprised ofhundreds of extent nodes, such as the extent node 310. In this example,the extent node serves extents in the stream layer 312. The extent nodesare dispersed across the domains by a fabric manager. In an exemplaryembodiment, the extent nodes are evenly dispersed across all domains ofa storage system. However, it is contemplated that extent nodes aredispersed through other methods (e.g., to satisfy demand). Additionally,in an exemplary embodiment, each of the replicas (i.e., copies) of anextent is located in different domains, which may be accomplished by astream manager.

Additionally, not shown at FIG. 3, but nonetheless contemplated, thestream layer 312 may also include a stream manager, which is a role thatmanages stream namespace, extents, and extent nodes in the stream layer312. In an exemplary embodiment, there are five to ten stream managersrole instances for a storage system. However, it is contemplated thatfewer or additional stream manager role instances may be implemented toachieve embodiments of the present invention. Additionally, in anexemplary embodiment, a fabric manager is responsible for dispersingeach stream manager role instance in a different domain. Additionally,the stream layer 312 may be comprised of a client library. The clientlibrary may be utilized to access the stream layer directly. Therefore,extent nodes, stream managers, and client libraries together providestream functionality.

FIG. 4 depicts a manager 400 in accordance with embodiment of thepresent invention. It is understood that the manager 400, in reality,may instead be a distributed operating system functional to control adistributed computing environment. Therefore, while the manager 400 isdiscussed herein as a single entity for discussion sake, the manager 400may instead be more abstract and distributed.

The manager 400 is comprised of components, which include: an upgradedomain selector 402, a notifying component 404, a data offloader/loader406, a monitoring component 408, a functional determining component 410,a role identifier 412, an instance quantity determining component 414,an extent identifier 416, a replica quantity determining component 418,a processor 420, a memory 422, a fabric manager 424, and a streammanager 426. It is understood that any component of the manager 400 mayutilize the processor 420 and/or the memory 422 to achieve one or morefunctionalities discussed herein. Additionally, it is contemplated thatany component may have a unique processor and/or memory that allow thecomponent to specially function to cause a transformation to achieveembodiments of the present invention.

The upgrade domain selector 402 is functional to select a domain from aplurality of domains. A domain is selected, in an exemplary embodiment,to be upgraded. Upgrading of a domain includes upgrading one or moreservices within the domain. For example, upgrading of a domain mayinclude upgrading a storage service, such as Windows Azure Storage, anexemplary storage service available from the Microsoft Corporation ofRedmond, Wash.

When a storage service is upgraded, durability and availability of dataare desired. Therefore, reliance on default mechanism that may beinherent to storage system failure recovery is not optimal for ananticipated or planned unavailability within the storage system. Forexample, a data system may be designed to recover and maintainavailability of data when a node fails; however, without preparing thedata on the node prior to failure, additional resource and/or additionaltime may be necessary to access the data of the failed node. Therefore,it is beneficial in an exemplary embodiment to implement embodiments ofthe present invention to overcome deficiencies of relying on inherentfailure recovery methods of a storage system to facilitate upgrading ofa node, service, or domain.

In an exemplary embodiment, the upgrade domain selector 402 isfunctional to identify a domain that is optimal for an initial upgrade.For example, the domain selector 402 may select a domain that is lightlyloaded or is utilizing a fewer number of resources. In an exemplaryembodiment, selection of an initial domain that has test data may bedesired in the event the upgrade is ineffective or causes one or morenodes of the domain to fail or not function properly. By selecting adomain with test data, less non-test (e.g., customer) data may thereforebe impacted by the ineffective upgrade. In an exemplary embodiment, theupgrade domain selector 402 is managed, at least in part, by the streammanager 426 when selecting a first domain to be upgraded.

Additionally, in an exemplary embodiment, the upgrade domain selector402 is functional to identify domains in different geo-related groups.For example, while domains may be serially upgraded within a commongeo-related group, domains that are in different geo-related groups fromone another may be upgraded in parallel. Therefore, the domain selector402 may identify domains that may be upgraded in parallel from differentgeo-related groups. Geo-related groups contain storage services thathave data being geo-replicated across the services. This geo-replicationcreates a dependency between the services; therefore, the services maynot be upgraded in parallel to prevent a failure potential during theupgrade.

In an exemplary embodiment, geo-replication within a geo-related groupmay be paused or temporarily suspended. In this example, thegeo-replication may be paused between a first cluster and a secondcluster that comprise a common geo-related group. For example, when thefirst cluster is being upgraded, geo-replication between the first andthe second cluster may be paused. This may prevent propagation of anerror to the second cluster that occurs during upgrade of the firstcluster. Following the upgrade of the first cluster, geo-replicationamong the first and the second cluster may resume. However, it iscontemplated that pausing geo-replication may result in thegeo-replication between the first cluster and the second cluster to getbehind schedule, which creates a chance of data loss if the firstcluster is to fail.

The notifying component 404 is functional to notify one or more devices,clients, services, nodes, or components of information related to adomain. For example, a domain or a portion of the domain may beunavailable as a result of an upgrade being performed. In this example,one or more nodes of a domain may be inaccessible during the process ofupgrading the domain; therefore, efficiencies may be realized inembodiment of the present invention if elements of an associateddistributed computing environment are notified of the upcomingavailability or unavailability of resources. As a result, the notifyingcomponent 404 provides notification of available or unavailable resourceto various elements of a distributed computing environment in anexemplary embodiment of the present invention.

The data offloader/loader 406 is functional to offload or load datafrom/to a node. In an exemplary embodiment, the data offloader/loader406 is functional to request data to be checkpointed and transferredfrom a node to another node. Checkpointing is a process of repackaging apartition with a modified state for faster loading. For example, whenfailure of a node that is serving a partition with a modified stateoccurs, the partition is re-loaded at the previous checkpoint and one ormore log files are relied upon to reconstruct the modified state thatexisted prior to failure of the node. In an exemplary embodiment,checkpointing of the partition allows for the modified state of thepartition to be integrated within the partition to reduce the resourcesrequired upon restart of a node. It is understood, that checkpointing ofdata includes requesting a node, device, or component to perform theaction of checkpointing. Therefore, the manager 400 including the fabricmanager 424 and the stream manager 426, in an embodiment, checkpoints,transfer, and/or offloads data from a node by requesting such an actionto be performed by the node or other associated elements.

Additionally, it is contemplated that the data offloader/loader 406 isfunctional to load data to a domain or associated nodes. For example,following an upgrade of a domain, partitions that were previouslyoffloaded to other domains may be re-loaded to the upgraded domain. Thisre-loading may be done, in part, to satisfy an affinity of the data orpartitions to a particular node or domain. In this example, the dataoffloader/loader 406 satisfies loading the data. In an exemplaryembodiment, a partition master may be referenced to aid in loading datato an affinitized node.

The monitoring component 408 is functional to monitoring a domain and/ornodes. The domain, in an exemplary embodiment, is monitored before,during, and/or after upgrading the domain. The monitoring component 408may employee any of a number of tests when monitoring a domain. Exampleof tests that may be used, which will be discussed in greater detailhereinafter, include: a smoke test, an availability test, a performancetest, an alert test, an error test, a dump test, a system resource test,and/or a system metadata health test. In general, tests used by themonitoring component 408 may be referred to herein as “health tests” asthe tests are generally evaluating the health of a domain, a role, aservice, or a node (e.g., health of a storage service). Further, it iscontemplated that any combination of tests, including those discussedherein and those similar in concept, may be implemented at variousstages of an upgrade to monitor the health of a domain.

The smoke test is a grouping of small tests that are designed to runquickly against parts of a storage system to ensure the system is in ahealthy state. The smoke test checks APIs associated with services,components, the fabric, and other software operating in the distributedcomputing environment to ensure they are operating as intended.Additionally, the smoke tests examine latency and throughputs ofoperations to ensure those values are at appropriate levels.Additionally, the smoke test verifies that API's, components, software,etc. are available and reachable as intended.

The availability test includes monitoring the availability of a role ora node within a domain. For example, during preparation or duringfinalization of an upgrade, if availability of the nodes within thedomain goes above or below a predefined threshold, the upgrade issuspended. Similar to other metrics associate with the health tests,availability is a metric that may be monitored by a storage manager or afabric manager regardless of an upgrade being implemented. It iscontemplated that monitoring is not limited to an upgrade domain butalso extends to other domains. For example, a first domain may beupgraded and the monitoring may indicate the first domain is healthy;however, another domain may be affected as a result of the upgrade tothe first domain. Monitoring may therefore be implemented system wide todetect ripple effects caused by an upgrade.

The performance test evaluates the latency and throughput of a domainand/or nodes within the domain. In an exemplary embodiment, theperformance test is implemented in conjunction with other tests thatdefine the smoke test. The performance test monitors the latency andthroughput, if latency increases above a predefined threshold orthroughput drops below a predefined threshold, the upgrade process issuspended or otherwise amended to ensure reliability, durability, andavailability of the a distributed computing environment.

The alert test monitors for alerts issued by nodes within a domain. Anode may provide an alert that internal errors, conflicts, or otherabnormalities have been detected, and as a result, an alert is providedthat is detected as part of the alert test.

The dump test monitors if crash dumps that occur within a domainincrease above a predefined threshold. Additionally, the error test alsomonitors the errors reported from a domain, such as from the nodeswithin a domain. Similar to the crash dumps, if the number of errorsincreases above a predefined threshold, the upgrade process may besuspended as an indication of trouble with the upgrade.

The system resource test monitors system resource utilization toidentify if the utilization increases above a predefined threshold. Forexample, the system resources may include CPU usage, memory usage,network usage, or the like.

The system metadata health test monitors and verifies that the keysystem metadata tables (e.g., schema tables, partition tables, and thelike), roles (e.g., partition managers and stream managers), and apredefined percentage of particular roles (e.g., front-end servers,partition servers, and extent node roles) are accessible and healthy.

The health tests may, as previously discussed, be implementedindependently or in any combination to monitor the health of a domain.Similarly, once a health test indicates a problem, a recent orconcurrent upgrade may be suspended from being implemented at otherdomains. Additionally, an upgrade may be “rolled-back” from a domain toreset the state of the domain to a previous position. For example, thedomain may revert to the previous “version” prior to an upgrade of aservice on the domain, or stated differently, the domain may bedowngraded.

Utilization of the health tests and monitoring the tests allows forautomated upgrading of the domains while maintaining availability,durability, and performance. This may be achieved by preventing thepropagation of faulty upgrades as identified by one or more healthtests. Therefore, services may continually be upgraded in an automatedfashion while ensuring that the distributed computing environment isresilient and is able to automatically adjust to detected abnormalitiesresulting from upgrades.

The functional determining component 410 is able to determine if adomain is functioning within predefined guidelines or satisfiespredefined conditions. For example, in conjunction with results fromhealth tests performed by the monitoring component 408, the functionaldetermining component 410 is able to identify when a domain is notfunctioning properly and therefore alter the progress of an upgrade. Inan exemplary embodiment, the functional determining component 410determines a domain is not operating appropriately and therefore theupgrade on that domain should be rolled-back and additionally, theupgrade should not be propagated to other domains. Further, it iscontemplated that previously upgraded domains may also be rolled-back.In an exemplary embodiment, predefined conditions and guidelines arebased on metrics associated with one or more health tests that aremonitored across the system as a whole. When those metrics aresatisfied, a domain is functioning properly, in an exemplary embodiment.

The role identifier 412 is functional to identify a role operatingwithin a domain. As previously discussed, a role is functionality or acomponent of a service. In an exemplary embodiment, the role identifier412 is able to identify a role associated with a service to which theupgrade is directed. In an additional exemplary embodiment, the roleidentifier 412 is functional to identify a plurality of roles operatingon a domain that is preparing to be upgraded. In this example, one ormore of the plurality of roles may be identified to ensure a propernumber of instances are running in domains other than the domain that isto be upgraded. Domains that are other than the upgrade domain may bereferred to herein as an “other domain.”

The instance quantity determining component 414 is functional todetermine a number or quantity of instances of a role operating in anupgrade domain and/or other domains. As will be discussed in more detaillater, embodiments of the present invention desire a predefined numberor relative percentage of instances to be operating in the otherdomains. For example, in this scenario, it is assumed that all instanceswithin the upgrade domain will be unavailable. While this may not betrue in actuality, for purposes of ensuring availability such asituation is assumed in this embodiment. Therefore, the instancequantity determining component 414 is able to identify a number ofinstances of an identified role that are operating and/or available inthe other domains.

The extent identifier 416 is functional to identify one or more extentsstored in a domain. For example, a domain may be comprised of aplurality of extent nodes (e.g., servers) that service a stream layer ofa storage service. In this example, to ensure durability, reliability,and availability, it may be desirable to have a predefined number ofreplicas of an extent distributed across the storage system. The extentidentifier 416 is able to identify one or more extents within an upgradedomain to ensure a sufficient quantity of replicas are available in theupgrade domain and/or other domains. In an exemplary embodiment, theextent identifier 416 is managed by and communicated with the streammanager 426 to provide described functionality.

The replica quantity determining component 418 is functional todetermine a number of replicas of an identified extent that areavailable and/or stored within an upgrade domain and/or other domains.For example, a domain may not be upgraded until a predefined number ofreplicas for each extent located at the upgrade domain are available atother domains. In an exemplary embodiment, a number of replicas may bedefined for specific domains to allow an upgrade to continue. As will bediscussed later, an upgrade may be suspended or cancelled until thestream layer has replicated the extent in other domains to a sufficientquantity. In an exemplary embodiment, replica quantity determiningcomponent 418 is managed by and communicated with the stream manager 426to provide described functionality.

The processor 420 is a computer processor functional to intakecomputer-readable instructions to output a transformation. The processor420, in an embodiment, is similar to the processor(s) 114 of FIG. 1.Similarly, the memory 422 is memory for use in a computing system. Thememory, in an embodiment, is similar to the memory 112 of FIG. 1.

The fabric manager 424 is a component functional to manage one or morecomponents of the manager 400 as well as to provide corefunctionalities. In an exemplary embodiment, the fabric manager 424manages components to accomplish management of nodes as well as knowinga state of upgrade or other status of the nodes. Further, the fabricmanager 424 manages components or provides functionality that are incharge of roles that are processes that make up a service. The fabricmanager 424 may control knowing the state of a role, controllingcommunication among roles, and restarting roles.

The stream manager 426 is a component functional to manage one or morecomponents of the manager 400. For example, the stream manager 426 maybe responsible for managing partitions, which may include partitionassignments, checkpointing, offloading, and location identification.Further, the stream manager 426 may also be responsible for managingextents, which may include extent instances and replication of extent.Further, the stream manager 426 may be responsible for managingcomponents to identify which extent instances are associated with whichextent nodes. For example, the stream manager may be responsible formanaging preparation and finalization stages of an upgrade to a domain.In an exemplary embodiment, the stream manager 426 is responsible formanaging the selection of the first domain to be updated. In anexemplary embodiment, the stream manager 426 manages the functionalityprovided by the offloader/loader 406, the instance quantity determiningcomponent 414, the extent identifier 416, and the replica quantitydetermining component 418. Further, in an exemplary embodiment, thestream manager 426 manages authorizing a partition node or upgradedomain to serve a partition. It is contemplated that managers similar tothe stream manager 426 may additionally be incorporated with the manager400 for service other than a storage service.

Accordingly, any number of components may be employed to achieve thedesired functionality within the scope of embodiments of the presentinvention. Although the various components of FIG. 4 are shown withlines for the sake of clarity, in reality, delineating variouscomponents is not so clear, and metaphorically, the lines would moreaccurately be gray or fuzzy. Further, although some components of FIG. 4are depicted as single blocks, the depictions are exemplary in natureand in number and are not to be construed as limiting.

FIG. 5 depicts a high-level method 500 for upgrading a domain inaccordance with embodiments of the present invention. In this exemplaryembodiment, the method 500 includes preparing to upgrade a domain 502,upgrading the domain 504, and finalizing the upgrade of the domain 506.In this example, the upgrading of a domain is comprised of three stepsthat help achieve efficiencies in upgrading a domain. To the contrary,if a domain of a distributed computing environment is intended to beupgraded and the domain is taken offline to perform the upgrade,inherent characteristics of a data store may allow the data store tocontinue to operate. However, an unanticipated failure of the domain, asa result of taking it offline and relying on inherent characteristic,fails to achieve the advantages as discussed herein. For example, whenrelying on inherent node failure recovery, a sufficient number of roleinstances may not exist nor are they provided time to be created,partitions may not be checkpointed for faster loading and greaterreliability, and a sufficient number of extent instances may not existnor are they provided an opportunity to be replicated in anticipation ofthe “failure.” These are just a few of the advantages that may beappreciated with embodiments of the present invention.

FIG. 6 depicts a diagram 600 illustrating a sequence for upgrading ofdomains in different geo-related groups in accordance with embodimentsof the present invention. A first group 602 and a second group 604,which are not geo-related to one another, are identified. Each of thegroups is comprised of a number of domains. For example, the first group602 includes a cluster that is comprised of a domain A 606, a domain B608, and a domain C 610. The second group 604 includes a cluster that iscomprised of a domain X 612 and a domain Y 614. Additionally, anabstract timeline is depicted that includes a T1 616, a T2 618, a T3620, and a T4 622.

A geo-related group is a grouping of storage clusters that are groupeddue to geo-replication dependencies. For example, a set of geo-relatedclusters may be comprised of clusters that geo-replicate among eachother. In this example, the clusters within a common geo-related groupare used to geo-replicate amongst each other. Therefore, in an exemplaryembodiment, multiple clusters in different geo-related groups may beupgraded in parallel, but the clusters within a given geo-related groupare upgraded in series. In addition embodiments, in a geo-related group,domains for a given cluster are upgraded before upgrading a next clusterwithin the geo-related group. Clusters are upgraded in series whenabstaining from upgrading a subsequent cluster until a prior cluster hasfinished upgrading. It is understood that any number of geo-relatedgroups may exists. Additionally, it is contemplated that the geo-relatedgroups may include clusters from a variety of geographic locations.Further, it is contemplated that a geo-related group is not limited by ageographic location, but instead, represents a grouping of clustersunified by a characteristic.

In an exemplary embodiment of the present invention, a singlegeo-related group exists, and therefore, the sequence may only includeserial upgrading. However, a number of geo-related groups may exist,which may therefore incorporate parallel upgrading among geo-relatedgroup and serial upgrading within each geo-related group.

For example, at T1 616 the domain A 606 is prepared for upgrade. In thisexample, a domain within the cluster of the second geo-related group 604may also begin upgrading at T1 616 because parallel upgrading amongdifferent geo-related group is feasible. However, in an exemplaryembodiment, an initial storage cluster or a subset of domains from thatstorage cluster is upgraded across all geo-related groups and storagecluster to reduce the spread of issues if the upgrade is unsuccessful.The initial cluster selected may be intelligently selected so that ithas wide spread traffic behavior and a minimal impact on non-test dataif there is a problem during upgrade. Therefore, the domain A 606 isupgraded initially to ensure the viability of the upgrade. As a result,the domain A 606 is prepared for upgrade, upgraded, and the upgrade isfinalized by time T2 618. During the upgrade process (i.e., prepare,upgrade, and finalize) health tests may be performed to monitor thehealth of the domain A 606. Once the domains A 606 is determined to befunctional by T2 618, additional domains may be upgraded.

In FIG. 6, the domains B 608 and the domain X 612 begin the upgrade attime T2 618. Therefore, the domains B 608 and the domain X 612 areupgrading in parallel. However, as depicted in FIG. 6, the domain B 608is completed with the upgrade at time T3 620, while the domain X 612 isnot completed with the upgrade until time T4 622. The time for anupgrade may depend on the number of resource available, the number ofinstance and extents affected, the amount of data to be checkpointed,and such. FIG. 6 demonstrates that domains within different geo-relatedgroups are not dependent upon one another for advancement of an upgrade.Additionally, FIG. 6 demonstrates that parallel upgrading may besuitable for clusters different geo-related groups and that serialupgrading is suitable within a geo-related group. However, FIG. 6 is notlimiting as to the scope of the present invention and is only anillustration of an exemplary embodiment.

FIG. 7 depicts a method 700 for upgrading a service in a distributedcomputing environment in accordance with embodiments of the presentinvention. The method 700 includes a step 702. At the step 702 anupgrade domain within a distributed computing environment is selected tobe upgraded. In an exemplary embodiment, a domain is upgraded when oneor more services that are supported by the domain, at least in part, areupgraded at the domain. Therefore, in this example, when a service thathas a role that is accessible in at least one node of a domain isupgraded, the domain is considered to be upgraded. Once a domain hasbeen selected to be upgraded, that domain may be referred to herein asan upgrade domain. An upgrade domain is a domain that includes at leastone instance of a role associated with a service that is intended to beupgraded.

The selection of the upgrade domain may be accomplished by a number ofmethodologies. For example, a random or pseudo random selection mayoccur. In an additional embodiment, the data type (e.g., test data,non-test data) within the domain may be used to select a domain.

At the step 704, data from a server in the upgrade domain is offloadedin anticipation of an upgrade. Therefore, unlike a random failure of thenode, an anticipated upgrade of the node within a domain allowspartitions to be checkpointed, which increases efficiencies withreloading the partition. In an exemplary embodiment, the offloading ofdata includes checkpointing one or more partitions that store the datain various extents. Additionally, in an exemplary embodiment, theoffloading of data includes transferring the data to one or more serversin a domain other than the upgrade domain. In this example, thetransferred data may therefore be accessible at the other domains whilethe upgrade domain is being upgraded. This helps ensure availability ofthe data. Further, checkpointing and transferring of the data from theupgrade domain allows redundancy level of the data to be maintained. Ifthe data was not transferred, as is the case when relying on inherentnode failure recovery, the redundancy levels may be compromised for thedata. In an additional exemplary embodiment, the offloading of dataincludes transferring the data in its current state to one or more otherdomains. Data that is offloaded includes roles, partitions, extents,and/or the like. It is contemplated that the transferring of data maynot include copying the data or moving the data. Instead, transferringof data may include re-assigning associated extents to a differentdomain or node than currently assigned. In an exemplary embodiment, areplication system allows for one instance of an extent to remain in adomain that is being upgraded without requiring replication. However, inthis example where one instance that remains on the upgrade domain, maynot have attempted accesses (e.g., avoiding reads), which may avoid atimeout associated with the access.

Offloading of data from a domain that is to be upgraded allows a currentlevel of data redundancy within the distributed computing environment tobe maintained. For example, a specific number of instances or replicasof roles and extents may be required. If inherent node failure recovertechniques are relied upon when upgrading a domain, the data of theupgraded domain may not be available and therefore the level ofredundancy of the data is reduced, which may increase the possibility ofunavailability or data loss. For this reason, embodiments of the presentinvention are advantageous over relying on inherent node failurerecovery techniques because a current level of redundancy is maintained.

At a step 706, a distributed computing environment is notified that theupgrade domain is unavailable. For example, a fabric manager may notifyor be notified that an upgrade domain is anticipated as beingunavailable as a result of an upgrade. In this example, the fabricmanager may provide notification to one or more clients that utilizepartitions, extents, or data in general that they are temporarilyunavailable from the upgrade domain. A client, such as a stream client,is an entity or service that requests information or data from a node.Because the client requests and expects to receive a response as aresult of the request, a service or entity in that position is referredto as a client.

In an exemplary embodiment, a service operating in the distributedcomputing environment may request data that is located, at least inpart, within an upgrade domain. However, because the fabric has provideda notification that the upgrade domain is unavailable; the request forthe data may be directed to a different domain that maintains anotherreplica of the requested information. By diverting the request from theupgrade domain to another viable domain, the system has preventedlatency or delays caused by an eventual time-out that would result fromthe unavailable upgrade domain. Therefore, notification that an upgradedomain is down is advantageous over inherent failure recovery techniquesthat may merely rely on the request timing out.

At a step 708, the service is upgraded. In an exemplary embodiment, aservice is upgraded by a fabric manager that facilitates the upgradingat one or more nodes. Therefore, upgrading the service includes a fabricmanager overseeing the operations done at one or more nodes that causethe service to be upgraded. A service, in an exemplary embodiment, isupgraded by updating one or more roles associated with the service. Theroles may be stored within the upgrade domain.

At a step 710, the data is loaded to the server. In an exemplaryembodiment, the data is loaded to the server from which it was offloadedat the step 704. Additionally, data may have an affinity to a specificdomain, node, or even data store. The loading of the data to the servermay take into consideration any affinities that the data may include.Further, in an exemplary embodiment, data is loaded to the server basedon one or more metrics, such as resource utilization, to mosteffectively distribute the load within a distributed computingenvironment.

At the step 712, the distributed computing environment is notified thatthe upgrade is complete. In an exemplary embodiment, one or more healthchecks are performed on the upgrade domain to ascertain the upgradedomain is functional and/or healthy; therefore, the upgrade domain isviable as a source/store for data. The notification that the upgradedomain is complete may indicate that one or more services, partitions,or extents are accessible by way of the upgrade domain. Therefore,requests for the data may be routed to the upgrade domain.

Turning to FIG. 8 that depicts a method 800 for upgrading a service in adistributed computing environment in accordance with embodiments of thepresent invention. The method 800 includes three overarching steps, eachof which may contain additional steps. For example, the step 802generally describes preparing an upgrade domain from an upgrade.However, a number of additional steps may further define the step 802.

The method 800 also includes a step 804. At the step 804, the upgradedomain is upgraded. Additionally, the method 800 includes a step 806. Atthe step 806, the upgrade domain is finalized.

The step 802 includes preparing an upgrade domain of the distributedcomputing environment for an upgrade. Preparing, at the step 802,includes steps 808-818. At the step 808, a role is identified. In anexemplary embodiment, the role is operating in the upgrade domain. Arole operates in the upgrade domain when it is accessible from theupgrade domain. As previously discussed, an instance of the role may belocated within other domains at any point. Therefore, at the step 810,an available quantity of instances of the role is determined. In anexemplary embodiment, the quantity of instances of the role isdetermined for domains other than the upgrade domain. This is, in part,to ensure that if all nodes of the upgrade domain were unavailableduring the upgrade or unable to recover from an upgrade, then asufficient quantity of the instances would be available at otherdomains.

A sufficient quantity of instance may depend on the particular instance.For example, a service defined by the role may dictate a number ofinstances that should be available to ensure availability andreliability. In an exemplary embodiment, the predefined threshold isdependent on the type of role. For example, a relative percentage offront-end nodes, partition nodes, and extent nodes may be required in anexemplary embodiment, For example, to ensure availability, 80% offront-end nodes, 80% of partition nodes, and 80% of extent nodes may bedemanded by a service to allow an upgrade to proceed. When thepredefined threshold is based on a relative percentage of availablenodes, this is a non-quorum-type role. Additionally, roles that insteadhave a predefined absolute quantity threshold are referred to asquorum-type roles.

Examples of quorum-type roles include the previously discussed partitionmanager, the stream manager, and a lock server system. A quorum-typerole is a role that necessitates a majority of instances to beavailable. In an exemplary embodiment, a quorum-type role is identifiedas being sufficient in quantity when there are at least a majority+1number. Therefore, if a quorum (or quorum+1 in an additional embodiment)does not exist in domains other than the upgrade domain, the upgrade maynot continue until the quorum is achieved. In an exemplary embodiment, afabric manager is functional to cause instances of roles to be createdto achieve a predefined threshold.

It is understood that a predefined threshold may be dynamically adjustedto achieve embodiment of the present invention. For example, during adowngrade (e.g., rollback) less stringent threshold demands may berequired in order to get the upgrade domain back into a functionalstate.

Additionally, during the step 802, front-end nodes in the upgrade domainmay be prevented from receiving new requests to allow the front-endnodes to complete outstanding requests. This process allows theoutstanding requests to be drained from a queue, which ensures higheravailability while upgrading the upgrade domain.

At the step 812, a partition in the upgrade domain is checkpointed. Aspreviously discussed, the checkpointing of the partition ensureavailability of the partition by allowing the current modified state ofthe partition to be captured and packed with the partition, which allowsa partition role to quickly load the partition.

At the step 814, an assignment to the partition is transferred to adomain other than the upgrade domain. In an exemplary embodiment, thedata comprising a partition is not moved because a partition layer isbuilt on top of a stream layer, which therefore acts as a distributedfile system. Therefore, in this example, an assignment to the partitionis transferred to another domain other than the upgrade domain. In anexemplary embodiment, the partition assignment is transferred to apartition node within a domain other than the current upgrade domain(e.g., a partition role running in a different domain). Further, in anexemplary embodiment, the recipient of the transferred partitionassignment begins serving request directed to the partition.

At the step 816, an extent in the upgrade domain is identified. Theextent is identified so that an available quantity of replicas of theextent in domains other than the upgrade domain may be determined to beabove a predefined threshold, as indicated at the step 818. Whenupgrading a domain, the nodes in the domain may need to be rebooted orthey may be taken offline for a period of time. Therefore, it isdesirable in this example to ensure that a sufficient number of replicasof the extents are available at domains other than the upgrade domain.Therefore, in this exemplary embodiment, when a sufficient number ofreplicas are not available, the upgrade is terminated or paused untiladditional replicas may be created by a stream layer. However, if asufficient number of replicas are found to exists, the upgrade maycontinue. A predefined threshold of replicas may be established at tworeplicas or three replicas in an exemplary embodiment. However, it iscontemplated that the predefined threshold may be any value to realizeembodiments of the present invention.

Upon completion of the steps 808-818, an upgrade domain is prepared foran upgrade in an exemplary embodiment. As a result, the upgrade domainmay then be upgraded as indicated at the step 804. The step 806 includesfinalizing the upgrade domain. Finalizing the upgrade domain may includea number of sub-steps as illustrated in the method 800. The sub-stepsinclude steps 820-826.

At the step 820, the partition is reloaded to a partition node. In anexemplary embodiment, an assignment to a partition is assigned to apartition node in order to effectuate the reloading of a partition.Further, in an exemplary embodiment, the partition is the partition thatwas transferred at step 814, plus any changes in the states thatoccurred in the interim. In an exemplary embodiment, the partition isreloaded from a checkpointed state. Therefore, the partition may beloaded quickly and efficiently. At the step 820, the partition node towhich the partition is reloaded may be selected based on an affinity ofthe partition.

At the step 822, the partition node is allowed to serve the partition.In an exemplary embodiment, a stream manager provides authorization tothe partition node or the upgrade domain to serve the partition. In anembodiment, the partition node is allowed to serve the partition as aresult of the stream manager updating one or more tables in thedistributed computing environment to indicate that the partition node isaccessible and/or the partition is stored thereon.

At the step 824, the distributed computing environment is notified thata node is available to store extents. In an exemplary embodiment, afabric manager provides a notification that the upgrade domain isavailable to store extents. This notification may be provided to thestream layer, which is typically responsible for storing extents.

At the step 826, a stream client is notified that a node is accessiblefor reading data. A fabric manager may notify a stream manager toprovide this notification by updating of tables or other resource that astream client would utilize to identify accessible nodes.

Additionally, as previously discussed, but not shown in FIG. 8, it iscontemplated that one or more health tests may be performed at any ofthe steps. For example, one or more health tests may be performed at thestep 802 and the step 806. Additionally, it is contemplated that at anyof the steps discussed herein, if a test fails to be satisfied or adetermination that a predefined threshold was not satisfied, the upgradeprocess may be stopped or temporally paused until a remedy may beimplemented.

Turning to FIG. 9 that depicts a method 900 for upgrading a service in adistributed computing environment in accordance with embodiments of thepresent invention. The method 900 includes a step 902, a step 904, and astep 906. The steps 902-906 are similar in concept to the steps 802-806of FIG. 8.

At the step 902 an upgrade domain that is comprised of a plurality ofservers of the distributed computing environment is prepared for anupgrade. Steps 908-918 are additional steps to achieve the step 902.

At the step 908, a number of instances of a role in one or more domainsother than the upgrade domain are determined to be above a predefinedthreshold. To the contrary, if the determination is that the predefinedthreshold is not achieved, the upgrade process may be suspended until ithas reached the predefined threshold. In an additional embodiment, theupgrade may be terminated. In the alternative, if the predefinedthreshold is not achieved, additional instances may be created to meetor exceed the predefined threshold. In an exemplary embodiment, the rolefor which a number of instances are being evaluated is responsible forat least some functionality associated with the service that is to beupgraded.

At the step 910, a partition served from a partition server in theupgrade domain is checkpointed to facilitate faster loading upontransfer of the partition to a domain other than the upgrade domain, asindicated at the step 912. At the step 912, an assignment to thepartition is transferred to the domain other than the upgrade domain.For example, a partition may be part of a partition layer which is builton top of a stream layer. In this example, the partition is not actuallymoved on media, but instead an assignment to the partition is reassignedto the other domain.

At the step 914, a number of replicas of an extent in one or moredomains other than the upgrade domain are determined to be above apredefined threshold. In the alternative, when the number of replicas isdetermined to be below the predefined threshold, the upgrade process maybe terminated or additional replicas are created at the stream layer. Inthis example, a stream manager may provide oversight to cause thereplication of extents to achieve the desired number. In an exemplaryembodiment, when the determined number of replicas is below a predefinedthreshold, the pending upgrade waits until the extent is replicated orother actions are taken to expedite the replication process.

At the step 916, a new extent is prevented from being created in theupgrade domain. As previously discussed, preventing new extents frombeing created may save the system resources by preventing the requestfor the creation of the extent from timing out or otherwise beingdelayed. Additionally, at the step 918, the distributed computingenvironment is notified that extents are unavailable from the upgradedomain. The notification may be provided by a stream manager, aspreviously discussed.

The steps 908-918 may be performed in any order and are not intended tobe limited to the order depicted and described herein. Similarly, FIG.7, FIG. 8, and FIG. 9 all depicts various method, none of which areintended to be limited to the order presented in an exemplaryembodiment.

At the step 904 the upgrade domain is upgraded with the upgrade. Aspreviously discussed, upgrading a domain includes upgrading a servicewithin the domain. At the step 908 the upgrade in the upgrade domain isfinalized. The method 900 includes a number of steps that define thestep 906. Steps 920-924 are additional steps that may be performed tofinalize an upgrade.

At the step 920, the partition server is authorized to serve one or morepartitions. A fabric manager may authorize the partition servers throughproviding notification that the partition server is accessible andcontains the partition. Therefore, authorization includes facilitatingaccess. In an additional embodiment, authorizing includes preventing aserver from serving a partition. Additionally, at the step 922, a newextent instance is allowed to be created in the upgrade domain.Similarly, the allowance may be provided by a manager in an exemplaryembodiment. Also, at the step 924 the distributed computing environmentis notified that one or more extents are available from the upgradedomain.

It is contemplated herein that one or more steps of a method may beimplemented by a computing device to achieve automatic or semi-automaticcompletion of a particular step or process. Further, it is contemplatedherein that a person may cause one or more steps to be implemented. Forexample, during an upgrade of a domain, the execution of an upgrade maybe initiated with human interaction. Additionally, it is contemplatedthat a computing device may cause the execution of an upgrade.Therefore, embodiments of the present invention may utilize humanintervention to facilitate realizing benefits of the embodiments.

Many different arrangements of the various components depicted, as wellas components not shown, are possible without departing from the spiritand scope of the present invention. Embodiments of the present inventionhave been described with the intent to be illustrative rather thanrestrictive. Alternative embodiments will become apparent to thoseskilled in the art that do not depart from its scope. A skilled artisanmay develop alternative means of implementing the aforementionedimprovements without departing from the scope of the present invention.

It will be understood that certain features and sub combinations are ofutility and may be employed without reference to other features and subcombinations and are contemplated within the scope of the claims. Notall steps listed in the various figures need be carried out in thespecific order described.

1. One or more computer storage media having computer-executableinstructions embodied thereon, that when executed by a computing systemhaving a processor and memory, cause the computing system to perform amethod for upgrading a service in a distributed computing environment,the method comprising: selecting an upgrade domain within thedistributed computing environment in which to upgrade the service;offloading data, in anticipation of the upgrade, from a first nodewithin the upgrade domain to one or more nodes in a domain other thanthe upgrade domain; notifying the distributed computing environment thatthe upgrade domain is unavailable; upgrading the service in the upgradedomain; loading the data to a second node in the upgrade domain; andnotifying the distributed computing environment that the upgrade domainis available.
 2. The computer storage media of claim 1, wherein themethod further comprises: monitoring the upgrade domain following theupgrading of the service; determining that the upgrade domain isfunctional following the upgrading of the service; and upon determiningthe upgrade domain is functional, selecting a subsequent upgrade domainwithin the distributed computing environment in which to upgrade theservice.
 3. The method of claim 2, wherein monitoring includesevaluating at least one of the following: a smoke test; an availabilitytest; a performance test; an alert test; an error test; a dump test; asystem resource test; or a system metadata health test.
 4. The method ofclaim 2, wherein the upgrade domain is functional when a monitored testachieves a predefined condition.
 5. The method of claim 1, wherein theupgrade domain is one of a first plurality of domains that define afirst geo-related group.
 6. The computer storage media of claim 5,wherein the method further comprises upgrading a domain in a secondgeo-related group in parallel with the upgrade domain.
 7. The computerstorage media of claim 5, wherein the method further comprises upgradinga subsequent domain in the first geo-related group in series with theupgrade domain, wherein upgrading the subsequent domain begins followingfinalization of the upgrade to the upgrade domain.
 8. The method ofclaim 1, wherein offloading data further comprises: checkpointing thedata; and transferring the data to one or more nodes in a domain otherthan the upgrade domain.
 9. The method of claim 1, wherein theoffloading data maintains a current level of data redundancy within thedistributed computing environment.
 10. A computer-implemented method forupgrading a service in a distributed computing environment, the methodcomprising: preparing an upgrade domain of the distributed computingenvironment for an upgrade, wherein preparing includes: identifying arole operating in the upgrade domain, determining that an availablequantity of instances of the role in domains other than the upgradedomain is above a predefined threshold, checkpointing a partition in theupgrade domain, transferring the partition to a domain other than theupgrade domain, identifying an extent in the upgrade domain, anddetermining that an available quantity of replicas of the extent indomains other than the upgrade domain is above a predefined threshold;and upgrading the upgrade domain with the upgrade.
 11. The method ofclaim 10 further comprising finalizing the upgrade domain.
 12. Themethod of claim 11, wherein finalizing the upgrade domain comprises:reloading the partition to a partition node in the upgrade domain; andallowing the partition node to serve the partition.
 13. The method ofclaim 11, wherein finalizing the upgrade domain comprises: notifying thedistributed computing environment that a node of the upgrade domain isavailable to store extents; and notifying a stream client in thedistributed computing environment that the node and one or moreassociated extent instances are accessible.
 14. The method of claim 10,wherein the predefined threshold for the available quantity of instancesof the role is a relative percentage of instances.
 15. The method ofclaim 10, wherein the predefined threshold for the available quantity ofinstances of the role is a number of instances.
 16. The method of claim10, wherein the predefined threshold for the available quantity ofreplicas is at least two replicas.
 17. The method of claim 10, whereinpreparing an upgrade domain further comprises, preventing, inanticipation of the upgrade, a front-end node of the upgrade domain fromreceiving a new request.
 18. The method of claim 17, wherein the roleoperating in the upgrade domain is a software component of the service.19. The method of claim 11 further comprising, abstaining from upgradingthe service at a domain other than the upgrade domain until the upgradeis finalized at the upgrade domain.
 20. One or more computer storagemedia having computer-executable instructions embodied thereon, thatwhen executed by a computing system having a processor and memory, causethe computing system to perform a method for upgrading a service in adistributed computing environment, the method comprising: preparing anupgrade domain comprised of a plurality of servers of the distributedcomputing environment for an upgrade, wherein preparing includes:determining that a number of instances of a role in one or more domainsother than the upgrade domain is above a predefined threshold, whereinthe role is responsible for at least some functionality associated withthe service, checkpointing, to facilitate loading upon transfer, apartition served from a partition server in the upgrade domain,transferring the partition to a domain other than the upgrade domain,determining that a number of replicas of an extent in one or moredomains other than the upgrade domain is above a predefined threshold,preventing a new extent instance from being created in the upgradedomain, and notifying the distributed computing environment that extentsare unavailable from the upgrade domain; upgrading the upgrade domainwith the upgrade; and finalizing the upgrade in the upgrade domain,wherein the finalizing includes: authorizing the partition server toserve one or more partitions, allowing a new extent instance to becreated in the upgrade domain, and notifying the distributed computingenvironment that one or more extent instances are available from theupgrade domain.